SeaDatabricksClient: Add Metadata Commands #593

varun-edachali-dbx · 2025-06-11T05:11:40Z

What type of PR is this?

Feature

Description

Add metadata command implementations for the SeaDatabricksClient (execution phase) - get_catalogs, get_schemas, get_tables and get_columns.

How is this tested?

Unit tests
E2E Tests
Manually - using the test scripts to be introduced in Introduce manual SEA test scripts for Exec Phase #589
N/A

The coverage of the functionality added (by test_filters.py and the new tests in test_sea_backend.py) are as below:

Module	Statements	Missing	Coverage	Notes
`filters.py`	33	1	97%	Line 21: `from databricks.sql.result_set import ResultSet, SeaResultSet` (TYPE_CHECKING import)
`sea/backend.py` (metadata methods)	121	0	100%	Fully covered

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-26T06:14:01Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-26T06:46:31Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-26T06:52:14Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jayantsing-db

LGTM. Added some questions.

jayantsing-db · 2025-06-26T06:55:36Z

src/databricks/sql/backend/sea/backend.py

+            session_id=session_id,
+            max_rows=max_rows,
+            max_bytes=max_bytes,
+            lz4_compression=False,


not using compression for metadata?

This is a side effect of setting use_cloud_fetch=False: compression is not supported for INLINE + JSON in SEA.

jayantsing-db · 2025-06-26T06:56:07Z

src/databricks/sql/backend/sea/backend.py

+            use_cloud_fetch=False,
+            parameters=[],
+            async_op=False,
+            enforce_embedded_schema_correctness=False,


this is a thrift-specific param?

Yes, but it is a param passed to Cursor's execute method, so I don't see a way to not include it in our execute_command, if it were passed as a connection level property then we may have been able to access it that way. Should I try to avoid passing it to the SEA backend?

src/databricks/sql/backend/sea/backend.py

jayantsing-db · 2025-06-26T07:01:05Z

src/databricks/sql/backend/sea/backend.py

+    ) -> "ResultSet":
+        """Get columns by executing 'SHOW COLUMNS IN CATALOG catalog [SCHEMA LIKE pattern] [TABLE LIKE pattern] [LIKE pattern]'."""
+        if not catalog_name:
+            raise ValueError("Catalog name is required for get_columns")


so for the caller (client) code, it will appear as ValueError. is it okay? should we throw something else as per spec? how does the other backend throws error in first-class APIs like these?

Thanks for the catch, I changed it to raise DatabaseError which seems the most appropriate, I also changed a few other methods to raise ProgrammingError instead of ValueError.

src/databricks/sql/backend/sea/utils/filters.py

jayantsing-db · 2025-06-26T07:12:09Z

src/databricks/sql/backend/sea/utils/filters.py

+        # Create a new ResultData object with filtered data
+        from databricks.sql.backend.sea.models.base import ResultData
+
+        result_data = ResultData(data=filtered_rows, external_links=None)


so when you manually set the result data in result set, what will happen in the fetch phase? will it use the execute-response to re-fetch the data? are there any other examples in the codebase, where we manually set the result data or is this the first instance?

Since we construct the ResultData by setting the data field and not external_links, the SeaResultSet is effectively instantiated as it would be during INLINE + JSON_ARRAY querying.

We also set it during the instantiation of the ResultSet in the SEA backend, as seen here.

src/databricks/sql/backend/sea/utils/filters.py

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-26T09:53:24Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

varun-edachali-dbx added 30 commits June 9, 2025 06:24

[squash from exec-sea] bring over execution phase changes

138c2ae

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess test

3e3ab94

Signed-off-by: varun-edachali-dbx <[email protected]>

add docstring

4a78165

Signed-off-by: varun-edachali-dbx <[email protected]>

remvoe exec func in sea backend

0dac4aa

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess files

1b794c7

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess models

da5a6fe

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess sea backend tests

686ade4

Signed-off-by: varun-edachali-dbx <[email protected]>

cleanup

31e6c83

Signed-off-by: varun-edachali-dbx <[email protected]>

re-introduce get_schema_desc

69ea238

Signed-off-by: varun-edachali-dbx <[email protected]>

remove SeaResultSet

66d7517

Signed-off-by: varun-edachali-dbx <[email protected]>

clean imports and attributes

71feef9

Signed-off-by: varun-edachali-dbx <[email protected]>

pass CommandId to ExecResp

ae9862f

Signed-off-by: varun-edachali-dbx <[email protected]>

remove changes in types

d8aa69e

Signed-off-by: varun-edachali-dbx <[email protected]>

add back essential types (ExecResponse, from_sea_state)

db139bc

Signed-off-by: varun-edachali-dbx <[email protected]>

fix fetch types

b977b12

Signed-off-by: varun-edachali-dbx <[email protected]>

excess imports

da615c0

Signed-off-by: varun-edachali-dbx <[email protected]>

reduce diff by maintaining logs

0da04a6

Signed-off-by: varun-edachali-dbx <[email protected]>

fix int test types

ea9d456

Signed-off-by: varun-edachali-dbx <[email protected]>

[squashed from exec-sea] init execution func

8985c62

Signed-off-by: varun-edachali-dbx <[email protected]>

remove irrelevant changes

d9bcdbe

Signed-off-by: varun-edachali-dbx <[email protected]>

remove ResultSetFilter functionality

ee9fa1c

Signed-off-by: varun-edachali-dbx <[email protected]>

remove more irrelevant changes

24c6152

Signed-off-by: varun-edachali-dbx <[email protected]>

remove more irrelevant changes

67fd101

Signed-off-by: varun-edachali-dbx <[email protected]>

even more irrelevant changes

271fcaf

Signed-off-by: varun-edachali-dbx <[email protected]>

remove sea response as init option

bf26ea3

Signed-off-by: varun-edachali-dbx <[email protected]>

exec test example scripts

ed7cf91

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

dae15e3

Signed-off-by: varun-edachali-dbx <[email protected]>

[squashed from sea-exec] merge sea stuffs

db5bbea

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess changes

d5d3699

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess removed docstring

6137a3d

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 23, 2025 08:40 — with GitHub Actions Failure

varun-edachali-dbx had a problem deploying to azure-prod June 24, 2025 11:06 — with GitHub Actions Failure

varun-edachali-dbx had a problem deploying to azure-prod June 24, 2025 15:48 — with GitHub Actions Failure

remove catalog requirement in get_tables

35f1ef0

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 26, 2025 01:57 — with GitHub Actions Failure

move filters.py to SEA utils

a515d26

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 26, 2025 05:38 — with GitHub Actions Failure

ensure SeaResultSet

59b1330

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 26, 2025 05:40 — with GitHub Actions Failure

varun-edachali-dbx added 2 commits June 26, 2025 05:48

Merge branch 'sea-migration' into metadata-sea

293e356

prevent circular imports

dd40beb

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 26, 2025 06:13 — with GitHub Actions Inactive

databricks deleted a comment from github-actions bot Jun 26, 2025

remove unused imports

14057ac

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 26, 2025 06:46 — with GitHub Actions Inactive

remove cast, throw error if not SeaResultSet

a4d5bdb

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 26, 2025 06:52 — with GitHub Actions Inactive

jayantsing-db approved these changes Jun 26, 2025

View reviewed changes

varun-edachali-dbx added 3 commits June 26, 2025 09:33

make SEA backend methods return SeaResultSet

e9b1314

Signed-off-by: varun-edachali-dbx <[email protected]>

use spec-aligned Exceptions in SEA backend

8ede414

Signed-off-by: varun-edachali-dbx <[email protected]>

remove defensive row type check

09a1b11

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 26, 2025 09:53 — with GitHub Actions Inactive

varun-edachali-dbx merged commit e380654 into sea-migration Jun 26, 2025
22 of 23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SeaDatabricksClient: Add Metadata Commands #593

SeaDatabricksClient: Add Metadata Commands #593

Uh oh!

varun-edachali-dbx commented Jun 11, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

jayantsing-db left a comment

Uh oh!

jayantsing-db Jun 26, 2025

Uh oh!

varun-edachali-dbx Jun 26, 2025

Uh oh!

jayantsing-db Jun 26, 2025

Uh oh!

varun-edachali-dbx Jun 26, 2025

Uh oh!

Uh oh!

jayantsing-db Jun 26, 2025

Uh oh!

varun-edachali-dbx Jun 26, 2025

Uh oh!

Uh oh!

jayantsing-db Jun 26, 2025

Uh oh!

varun-edachali-dbx Jun 26, 2025

Uh oh!

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

SeaDatabricksClient: Add Metadata Commands #593

SeaDatabricksClient: Add Metadata Commands #593

Uh oh!

Conversation

varun-edachali-dbx commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Description

How is this tested?

Related Tickets & Documents

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

jayantsing-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

varun-edachali-dbx commented Jun 11, 2025 •

edited

Loading